United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 

Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria. Virginia 22313-1450 
www.uspto.gov 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. 



CONFIRMATION NO. 



10/680,032 



10/06/2003 



Marcio dos Galli 



22862 7590 10/ 

GLENN PATENT GROUP 
3475 EDISON WAY, SUITE L 
MENLO PARK, CA 94025 



BATURAY, ALICIA 



PAPER NUMBER 



NOTIFICATION DATE 



DELIVERY MODE 



10/26/2009 ELECTRONIC 

Please find below and/or attached an Office communication concerning this application or proceeding. 

The time period for reply, if any, is set in the attached communication. 

Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the 
following e-mail address(es): 

eptomatters@glenn-law.com 



PTOL-90A (Rev. 04/07) 



Office Artinn Qn mmarw 


Application No. 

10/680,032 


Applicant(s) 
GALLI ET AL. 


Examiner 

Alicia Baturay 


Art Unit 

2446 





-- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )E3 Responsive to communication(s) filed on 19 June 2009 . 
2a)^ This action is FINAL. 2b)Q This action is non-final. 

3) \Z\ Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) ^3 Claim(s) 1,2,4-10.12-26 and 28-45 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) Q Claim(s) is/are allowed. 

6) E3 Claim(s) 1,2,4-10.12-26 and 28-45 is/are rejected. 

7) 0 Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) ^ The drawing(s) filed on 06 October 2003 is/are: a)^ accepted or b)Q objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) Q Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)D All b)Q Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

20 Certified copies of the priority documents have been received in Application No. . 

30 Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attach ment(s) 

1 ) ^ Notice of References Cited (PTO-892) 

2) Q Notice of Draftsperson's Patent Drawing Review (PTO-948) 

3) □ Information Disclosure Statement(s) (PTO/SB/08) 

Paper No(s)/Mail Date . 



4) □ Interview Summary (PTO-41 3) 

Paper No(s)/Mail Date. . 

5) O Notice of Informal Patent Application 

6) □ Other: . 



U.S. Patent and Trademark Office 

PTOL-326 (Rev. 08-06) 



Office Action Summary 



Part of Paper No./Mail Date 20091020 



Application/Control Number: 10/680,032 Page 2 

Art Unit: 2446 

DETAILED ACTION 

1. The instant application has been assigned to a new examiner. Please see conclusion 
section for updated contact information. 

2. This Office Action is in response to the amendment filed 19 June 2009. 

3. Claims 1, 2, 4, 5, 25 and 26 were amended. 

4. Claims 3,11 and 27 were cancelled. 

5. Claims 34-45 were added. 

6. Claims 1, 2, 4-10, 12-26 and 28-45 are pending in this Office Action. 

Response to Amendment 

7. The objection to claim 26 regarding minor informalities was addressed and is withdrawn. 

8. Applicant's amendments and arguments with respect to claims 1, 2, 4-10, 12-26 and 28- 
33 and new claims 34-45 filed on 19 June 2009 have been fully considered but they are 
deemed to be moot in view of the new grounds of rejection. 

Claim Rejections - 35 USC § 103 

9. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 
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10. Claims 1, 2, 4, 12, 25, 26 and 28 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Yairi et al. (U.S. 2004/0078424) in view of Kusuda (U.S. 2003/0088623) 
and further in view of Seme (U.S. 2003/0125927). 



Yairi teaches the invention substantially as claimed including a method and system for 
accessing one or more web services (WS) from a mobile terminal using an instant messaging 
(IM) client are provided. When the IM client requests to communicate with a web service 
virtual user, the IM message is routed through a mobile IM server to an IM/WS gateway, 
which obtains a description of the requested web service, prompts the IM client for any 
required web service input, and composes a web services formatted message to send to the 
web services provider. When the IM/WS gateway receives a response back from the web 
service, the IM/WS gateway translates the response into one or more IM messages and sends 
the IM message(s) to the requester IM client. The IM/WS gateway can combine web services 
to provide a higher value service to an IM user (see Abstract). 



1 1 . With respect to claim 1 , Yairi teaches a system for providing real-time communication 
over a global network in a session between two or more users, each of said two or more users 
using a device communicatively coupled to the global network (Yairi, page 2, paragraph 23), 
said system comprising: a client messaging application which runs on each of a plurality of 
user devices, at least two of said user devices including a user device screen, said client 
messaging application providing a user interface displayed on each of said user device 
screens (Yairi, page 2, paragraph 23), said user interface comprising: a message entry 
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window for a user to enter data (Yairi, Fig. 8B); a communication window for displaying 
messages entered in said session (Yairi, Fig. 8B); and a selection window for accessing one 
or more application agents, associated to an external application (Yairi, pages 1-2, paragraph 
9 and page 4, paragraph 38); thus providing an enriched communication session beyond 
simple, replicated text message content (Yairi, pages 1-2, paragraph 9 and page 4, paragraphs 
38-39; GPS, phonebook, calendar, web browser, email, stock ticker information); wherein 
when one of said one or more application agents is activated, said external application 
represented by said one or more activated application agents (Yairi, page 2, paragraph 10; 
page 4, paragraph 33; page 5, paragraph 40; the web services). 

Yairi does not explicitly teach one or more activated application agents is displayed on 
said user device screens in conjunction with said client messaging application. 

However, Kusuda teaches one or more activated application agents is activated to run in 
conjunction with said client messaging application such that said two or more users in said 
session can utilize said external application without leaving said session (Kusuda, page 1, 
paragraph 7; page 2, paragraph 18-21). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify Yairi in view of Kusuda in order to enable one or more activated 
application agents is displayed on said user device screens in conjunction with said client 
messaging application. One would be motivated to do so in order to provide collaboration 
between two users on one external application (Kusuda, page 1, paragraphs 10 and 11). 

The combination of Yairi and Kusuda does not explicitly teach translating content as it is 
typed into said message entry window. 
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However, Seme teaches a connection to said translation service application, said 
translation service application capable of translating content as it is typed into said message 
entry window (Seme, page 3, paragraphs 26, 28 and 29). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi and Kusuda in view of Seme in order to enable 
translating content as it is typed into said message entry window. One would be motivated to 
do so in order to instantly translate real-time messages such that multilingual communication 
is achieved between one or more computing devices without compromising the real-time user 
experience (Seme, page 1, paragraph 5). 

12. With respect to claim 2, the combination of Yairi, Kusuda and Seme teaches the 
invention described in claim 1, including the system wherein said translation service 
application is either a local application in any of said user's device (Seme, page 3, paragraph 
25) or a service on the global network (Seme, page 3, paragraph 27). 

13. With respect to claim 4, the combination of Yairi, Kusuda and Seme teaches the 
invention described in claim 1 , including the system wherein said activated application agent 
addresses said translation service application on an other side of said session via the user's 
message protocol (Seme, page 3, paragraph 27). 

14. With respect to claim 12, the combination of Yairi, Kusuda and Seme teaches the 
invention described in claim 1, including the system wherein said selection window displays 
any of: a list of unregistered application agents, each of said unregistered application agents 
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being available to be registered with said client messaging application (Yairi, Figs. 4 and 8A; 
page 4, paragraph 39); and a list of registered application agents, each of said registered 
application agents being immediately available to be activated by a user (Yairi, Figs. 4 and 
8 A; page 4, paragraph 39). 

15. With respect to claim 25, Yairi teaches a method for incorporating external resources into 
an instant messaging session supported by an instant messaging system (Yairi, page 2, 
paragraph 23), said instant messaging system comprising a client messaging application 
which runs on devices communicatively coupled to the Internet (Yairi, page 2, paragraph 23), 
comprising the steps of: providing through said client messaging application a user interface 
displayed on each device's screen from which a user communicates with another user (Yairi, 
page 2, paragraph 23), said user interface comprising a message entry window for said user 
to enter data (Yairi, Fig. 8B), a communication window for displaying messages entered in 
said instant messaging session (Yairi, Fig. 8B), and a selection window for accessing one or 
more application agents, each of said application agents being associated with an external 
application (Yairi, pages 1 and 2, paragraph 9, and page 4, paragraph 38), the method further 
comprising the steps of: activating an application agent from a list of available application 
agents, wherein each of said application agents is available to be activated by said user 
(Yairi, page 2, paragraph 10 and page 4, paragraph 33), thereby activating the external 
application to which said activated application agent is associated, said external application 
capable of providing an enriched communication session beyond simple, replicated text 
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message content (Yairi, pages 1 and 2, paragraph 9, and page 4, paragraphs 38 and 39; GPS, 
phonebook, calendar, web browser, email, stock ticker information). 
Yairi does not explicitly teach sharing said external application. 

However, Kusuda teaches sharing said external application between at least two users in 
said instant messaging session (Kusuda, page 1, paragraph 7, and page 2, paragraphs 18-21). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify Yairi in view of Kusuda in order to enable sharing said external 
application. One would be motivated to do so in order to provide collaboration between two 
users on one external application (Kusuda, page 1, paragraphs 10 and 11). 

The combination of Yairi and Kusuda does not explicitly teach translating content as it is 
typed into said message entry window. 

However, Seme teaches said translation service application capable of translating content 
as it is typed into said message entry window (Seme, page 3, paragraphs 26, 28 and 29). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi and Kusuda in view of Seme in order to enable 
translating content as it is typed into said message entry window. One would be motivated to 
do so in order to instantly translate real-time messages such that multilingual communication 
is achieved between one or more computing devices without compromising the real-time user 
experience (Seme, page 1, paragraph 5). 

16. With respect to claim 26, the combination of Yairi, Kusuda and Seme teaches the 
invention described in claim 25, including the method wherein said external application is 
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either a local application in any of said devices communicatively coupled to the Internet or a 
service on the Internet (Yairi, page 2, paragraph 23). 

17. With respect to claim 28, the combination of Yairi, Kusuda and Seme teaches the 
invention described in claim 25, including the method wherein at least one of said registered 
application agents is associated to an interactive service (Yairi, pages 1-2, paragraph 9, and 
page 3, paragraph 25). 



18. Claim 5 is rejected under 35 U.S.C. 103(a) as being unpatentable over Yairi in view of 
Kusuda in view of Seme and further in view of McMullin et al. (US 2004/0125924). 

19. With respect to claim 5, Yairi teaches the invention described in claim 1, including a 
system for providing real-time communication over a global network in a session between 
two or more users, each of said two or more users using a device communicatively coupled 
to the global network (Yairi, page 2, paragraph 23), said system comprising: a client 
messaging application which runs on each of a plurality of user devices, at least two of said 
user devices including a user device screen, said client messaging application providing a 
user interface displayed on each of said user device screens (Yairi, page 2, paragraph 23), 
said user interface comprising: a message entry window for a user to enter data (Yairi, Fig. 
8B); a communication window for displaying messages entered in said session (Yairi, Fig. 
8B); and a selection window for accessing one or more application agents, associated to an 
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external application (Yairi, pages 1-2, paragraph 9 and page 4, paragraph 38); thus providing 
an enriched communication session beyond simple, replicated text message content (Yairi, 
pages 1-2, paragraph 9 and page 4, paragraphs 38-39; GPS, phonebook, calendar, web 
browser, email, stock ticker information); wherein when one of said one or more application 
agents is activated, said external application represented by said one or more activated 
application agents (Yairi, page 2, paragraph 10; page 4, paragraph 33; page 5, paragraph 40; 
the web services). 

Yairi does not explicitly teach one or more activated application agents is displayed on 
said user device screens in conjunction with said client messaging application. 

However, Kusuda teaches one or more activated application agents is activated to run in 
conjunction with said client messaging application such that said two or more users in said 
session can utilize said external application without leaving said session (Kusuda, page 1, 
paragraph 7; page 2, paragraph 18-21). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify Yairi in view of Kusuda in order to enable one or more activated 
application agents is displayed on said user device screens in conjunction with said client 
messaging application. One would be motivated to do so in order to provide collaboration 
between two users on one external application (Kusuda, page 1, paragraphs 10 and 11). 

The combination of Yairi and Kusuda does not explicitly teach translating content as it is 
typed into said message entry window. 
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However, Seme teaches a connection to said translation service application, said 
translation service application capable of translating content as it is typed into said message 
entry window (Seme, page 3, paragraphs 26, 28 and 29). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi and Kusuda in view of Seme in order to enable 
translating content as it is typed into said message entry window. One would be motivated to 
do so in order to instantly translate real-time messages such that multilingual communication 
is achieved between one or more computing devices without compromising the real-time user 
experience (Seme, page 1, paragraph 5). 

The combination of Yairi, Kusuda and Seme does not explicitly teach data transferred via 
said message protocol for addressing said translation service application on said other side of 
said session is a sequence of characters that represents binary. 

However, McMullin teaches data transferred via said message protocol for addressing 
said external application on said other side of said session is a sequence of characters that 
represents binary (McMullin, page 3, paragraph 34). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi, Kusuda and Seme in view of McMullin in 
order to enable data transferred via said message protocol for addressing said translation 
service application on said other side of said session is a sequence of characters that 
represents binary. One would be motivated to do so in order to send data through a digital 
communications network (McMullin, page 3, paragraph 34). 
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20. Claims 6, 31 and 33 are rejected under 35 U.S.C. 103(a) as being unpatentable over Yairi 
in view of Kusuda in view of Seme and further in view of Bjoernsen et al. (U.S. 
2004/0174392). 

21. With respect to claim 6, Yairi teaches the invention described in claim 1, including a 
system for providing real-time communication over a global network in a session between 
two or more users, each of said two or more users using a device communicatively coupled 
to the global network (Yairi, page 2, paragraph 23), said system comprising: a client 
messaging application which runs on each of a plurality of user devices, at least two of said 
user devices including a user device screen, said client messaging application providing a 
user interface displayed on each of said user device screens (Yairi, page 2, paragraph 23), 
said user interface comprising: a message entry window for a user to enter data (Yairi, Fig. 
8B); a communication window for displaying messages entered in said session (Yairi, Fig. 
8B); and a selection window for accessing one or more application agents, associated to an 
external application (Yairi, pages 1-2, paragraph 9 and page 4, paragraph 38); thus providing 
an enriched communication session beyond simple, replicated text message content (Yairi, 
pages 1-2, paragraph 9 and page 4, paragraphs 38-39; GPS, phonebook, calendar, web 
browser, email, stock ticker information); wherein when one of said one or more application 
agents is activated, said external application represented by said one or more activated 
application agents (Yairi, page 2, paragraph 10; page 4, paragraph 33; page 5, paragraph 40; 
the web services) and wherein said selection window for accessing one or more application 
agents further comprises: one or more distinct visual cues, each of which being representative 
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of one or more said application agents (Yairi, Fig. 4 and 8a; page 4, paragraph 39); and said 
one or more distinct visual cues shown in association with contact in a contact list of a user 
(Yairi, Fig. 4 and 8a; page 4, paragraph 39). 

Yairi does not explicitly teach one or more activated application agents is displayed on 
said user device screens in conjunction with said client messaging application. 

However, Kusuda teaches one or more activated application agents is activated to run in 
conjunction with said client messaging application such that said two or more users in said 
session can utilize said external application without leaving said session (Kusuda, page 1, 
paragraph 7; page 2, paragraph 18-21). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify Yairi in view of Kusuda in order to enable one or more activated 
application agents is displayed on said user device screens in conjunction with said client 
messaging application. One would be motivated to do so in order to provide collaboration 
between two users on one external application (Kusuda, page 1, paragraphs 10 and 11). 

The combination of Yairi and Kusuda does not explicitly teach translating content as it is 
typed into said message entry window. 

However, Seme teaches a connection to said translation service application, said 
translation service application capable of translating content as it is typed into said message 
entry window (Seme, page 3, paragraphs 26, 28 and 29). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi and Kusuda in view of Seme in order to enable 
translating content as it is typed into said message entry window. One would be motivated to 
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do so in order to instantly translate real-time messages such that multilingual communication 
is achieved between one or more computing devices without compromising the real-time user 
experience (Seme, page 1 , paragraph 5). 

The combination of Yairi, Kusuda and Seme does not explicitly teach associating 
contacts based on prior use. 

However, Bjoernsen teaches an association based on said user's prior use of said 
application agents with said contact (Bjoernsen, Fig. 10; page 1, paragraph 6). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi, Kusuda and Seme in view of Bjoernsen in 
order to enable associating contacts based on prior use. One would be motivated to do so in 
order to include a buddy list with popular and frequent contacts in order to allow 
collaboration between users over instant messaging services (Bjoernsen, page 1, paragraphs 2 
and 4). 

22. With respect to claim 31, Yairi teaches the invention described in claim 25, including a 
method for incorporating external resources into an instant messaging session supported by 
an instant messaging system (Yairi, page 2, paragraph 23), said instant messaging system 
comprising a client messaging application which runs on devices communicatively coupled 
to the Internet (Yairi, page 2, paragraph 23), comprising the steps of: providing through said 
client messaging application a user interface displayed on each device's screen from which a 
user communicates with another user (Yairi, page 2, paragraph 23), said user interface 
comprising a message entry window for said user to enter data (Yairi, Fig. 8B), a 
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communication window for displaying messages entered in said instant messaging session 
(Yairi, Fig. 8B), and a selection window for accessing one or more application agents, each 
of said application agents being associated with an external application (Yairi, pages 1 and 2, 
paragraph 9, and page 4, paragraph 38), the method further comprising the steps of: 
activating an application agent from a list of available application agents, wherein each of 
said application agents is available to be activated by said user (Yairi, page 2, paragraph 10 
and page 4, paragraph 33), thereby activating the external application to which said activated 
application agent is associated, said external application capable of providing an enriched 
communication session beyond simple, replicated text message content (Yairi, pages 1 and 2, 
paragraph 9, and page 4, paragraphs 38 and 39; GPS, phonebook, calendar, web browser, 
email, stock ticker information) and associating one or more of visual cues to a contact in a 
contact list of a user (Yairi, Figs. 4 and 8 A; page 4, paragraph 39), each of said one or more 
visual cues representing one of said registered application agents (Yairi, Figs. 4 and 8A; page 
4, paragraph 39). 

Yairi does not explicitly teach sharing said external application. 

However, Kusuda teaches sharing said external application between at least two users in 
said instant messaging session (Kusuda, page 1, paragraph 7, and page 2, paragraphs 18-21). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify Yairi in view of Kusuda in order to enable sharing said external 
application. One would be motivated to do so in order to provide collaboration between two 
users on one external application (Kusuda, page 1, paragraphs 10 and 11). 
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The combination of Yairi and Kusuda does not explicitly teach translating content as it is 
typed into said message entry window. 

However, Seme teaches said translation service application capable of translating content 
as it is typed into said message entry window (Seme, page 3, paragraphs 26, 28 and 29). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi and Kusuda in view of Seme in order to enable 
translating content as it is typed into said message entry window. One would be motivated to 
do so in order to instantly translate real-time messages such that multilingual communication 
is achieved between one or more computing devices without compromising the real-time user 
experience (Seme, page 1, paragraph 5). 

The combination of Yairi, Kusuda and Seme does not explicitly teach associating based 
on prior use or frequency. 

However, Bjoernsen teaches association based on said user's prior use or use frequency, 
with said contact, of said registered application agents (Bjoernsen, Fig. 10; page 1, paragraph 
6). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi, Kusuda and Seme in view of Bjoernsen in 
order to enable associating based on prior use or frequency. One would be motivated to do so 
in order to include a buddy list with popular and frequent contacts in order to allow 
collaboration between users over instant messaging services (Bjoernsen, page 1, paragraphs 2 
and 4). 
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23. With respect to claim 33, the combination of Yairi, Kusuda, Seme and Bjoemsen teaches 
the invention described in claim 31, including the method wherein said contact can be any of: 
a screen name representing a human contact (Yairi, Fig. 8B); a name or a visual cue 
representing an interactive service; and a name or a visual cue representing one of said 
registered application agents. 

24. Claims 7-10 and 32 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Yairi in view of Kusuda in view of Seme in view of Bjoernsen and further in view of 
Dickerman et al. (U.S. 2003/0177184). 

25. With respect to claim 7, Yairi teaches the invention described in claim 6, including a 
system for providing real-time communication over a global network in a session between 
two or more users, each of said two or more users using a device communicatively coupled 
to the global network (Yairi, page 2, paragraph 23), said system comprising: a client 
messaging application which runs on each of a plurality of user devices, at least two of said 
user devices including a user device screen, said client messaging application providing a 
user interface displayed on each of said user device screens (Yairi, page 2, paragraph 23), 
said user interface comprising: a message entry window for a user to enter data (Yairi, Fig. 
8B); a communication window for displaying messages entered in said session (Yairi, Fig. 
8B); and a selection window for accessing one or more application agents, associated to an 
external application (Yairi, pages 1-2, paragraph 9 and page 4, paragraph 38); thus providing 
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an enriched communication session beyond simple, replicated text message content (Yairi, 
pages 1-2, paragraph 9 and page 4, paragraphs 38-39; GPS, phonebook, calendar, web 
browser, email, stock ticker information); wherein when one of said one or more application 
agents is activated, said external application represented by said one or more activated 
application agents (Yairi, page 2, paragraph 10; page 4, paragraph 33; page 5, paragraph 40; 
the web services) and wherein said selection window for accessing one or more application 
agents further comprises: one or more distinct visual cues, each of which being representative 
of one or more said application agents (Yairi, Fig. 4 and 8a; page 4, paragraph 39); and said 
one or more distinct visual cues shown in association with contact in a contact list of a user 
(Yairi, Fig. 4 and 8a; page 4, paragraph 39). 

Yairi does not explicitly teach one or more activated application agents is displayed on 
said user device screens in conjunction with said client messaging application. 

However, Kusuda teaches one or more activated application agents is activated to run in 
conjunction with said client messaging application such that said two or more users in said 
session can utilize said external application without leaving said session (Kusuda, page 1, 
paragraph 7; page 2, paragraph 18-21). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify Yairi in view of Kusuda in order to enable one or more activated 
application agents is displayed on said user device screens in conjunction with said client 
messaging application. One would be motivated to do so in order to provide collaboration 
between two users on one external application (Kusuda, page 1, paragraphs 10 and 11). 



Application/Control Number: 10/680,032 Page 18 

Art Unit: 2446 

The combination of Yairi and Kusuda does not explicitly teach translating content as it is 
typed into said message entry window. 

However, Seme teaches a connection to said translation service application, said 
translation service application capable of translating content as it is typed into said message 
entry window (Seme, page 3, paragraphs 26, 28 and 29). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi and Kusuda in view of Seme in order to enable 
translating content as it is typed into said message entry window. One would be motivated to 
do so in order to instantly translate real-time messages such that multilingual communication 
is achieved between one or more computing devices without compromising the real-time user 
experience (Seme, page 1 , paragraph 5). 

The combination of Yairi, Kusuda and Seme does not explicitly teach associating 
contacts based on prior use. 

However, Bjoernsen teaches an association based on said user's prior use of said 
application agents with said contact (Bjoernsen, Fig. 10; page 1, paragraph 6). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi, Kusuda and Seme in view of Bjoernsen in 
order to enable associating contacts based on prior use. One would be motivated to do so in 
order to include a buddy list with popular and frequent contacts in order to allow 
collaboration between users over instant messaging services (Bjoernsen, page 1, paragraphs 2 
and 4). 
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The combination of Yairi, Kusuda, Seme and Bjoernsen does not explicitly teach the 
agent automatically activated. 

However, Dickerman teaches wherein whenever said contact joins said session, said 
application agents represented by said one or more distinct visual cues associated with said 
contact are automatically activated (Dickerman, pages 6 and 7, paragraphs 34-36). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi, Kusuda, Seme and Bjoernsen in view of 
Dickerman in order to enable the agent automatically activated. One would be motivated to 
do so in order to include a buddy list with popular and frequent contacts in order to 
automatically activating an agent when a user joins in order to allow a user to invite other 
users to collaborate with the registered application (Dickerman, page 7, paragraph 36). 

26. With respect to claim 8, the combination of Yairi, Kusuda, Seme, Bjoernsen and 
Dickerman teaches the invention described in claim 6, including wherein any of said 
application agents can be registered as a contact in said contact list of said user to create 
registered application agents (Dickerman, page 7, paragraph 36). 

27. With respect to claim 9, the combination of Yairi, Kusuda, Seme, Bjoernsen and 
Dickerman teaches the invention described in claim 8, including wherein at least one of said 
registered application agents is associated to an interactive service (Yairi, pages 1 and 2, 
paragraph 9 and page 3, paragraph 25). 
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28. With respect to claim 10, the combination of Yairi, Kusuda, Seme, Bjoernsen and 
Dickerman teaches the invention described in claim 9, including wherein one or more 
registered application agents can be run in conjunction with said interactive service 
associated with said at least one registered application agent (Dickerman, page 7, paragraph 
36). 

29. With respect to claim 32, Yairi teaches the invention described in claim 31, including a 
method for incorporating external resources into an instant messaging session supported by 
an instant messaging system (Yairi, page 2, paragraph 23), said instant messaging system 
comprising a client messaging application which runs on devices communicatively coupled 
to the Internet (Yairi, page 2, paragraph 23), comprising the steps of: providing through said 
client messaging application a user interface displayed on each device's screen from which a 
user communicates with another user (Yairi, page 2, paragraph 23), said user interface 
comprising a message entry window for said user to enter data (Yairi, Fig. 8B), a 
communication window for displaying messages entered in said instant messaging session 
(Yairi, Fig. 8B), and a selection window for accessing one or more application agents, each 
of said application agents being associated with an external application (Yairi, pages 1 and 2, 
paragraph 9, and page 4, paragraph 38), the method further comprising the steps of: 
activating an application agent from a list of available application agents, wherein each of 
said application agents is available to be activated by said user (Yairi, page 2, paragraph 10 
and page 4, paragraph 33), thereby activating the external application to which said activated 
application agent is associated, said external application capable of providing an enriched 
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communication session beyond simple, replicated text message content (Yairi, pages 1 and 2, 
paragraph 9, and page 4, paragraphs 38 and 39; GPS, phonebook, calendar, web browser, 
email, stock ticker information) and associating one or more of visual cues to a contact in a 
contact list of a user (Yairi, Figs. 4 and 8A; page 4, paragraph 39), each of said one or more 
visual cues representing one of said registered application agents (Yairi, Figs. 4 and 8A; page 
4, paragraph 39). 

Yairi does not explicitly teach sharing said external application. 

However, Kusuda teaches sharing said external application between at least two users in 
said instant messaging session (Kusuda, page 1, paragraph 7, and page 2, paragraphs 18-21). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify Yairi in view of Kusuda in order to enable sharing said external 
application. One would be motivated to do so in order to provide collaboration between two 
users on one external application (Kusuda, page 1, paragraphs 10 and 11). 

The combination of Yairi and Kusuda does not explicitly teach translating content as it is 
typed into said message entry window. 

However, Seme teaches said translation service application capable of translating content 
as it is typed into said message entry window (Seme, page 3, paragraphs 26, 28 and 29). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi and Kusuda in view of Seme in order to enable 
translating content as it is typed into said message entry window. One would be motivated to 
do so in order to instantly translate real-time messages such that multilingual communication 
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is achieved between one or more computing devices without compromising the real-time user 
experience (Seme, page 1, paragraph 5). 

The combination of Yairi, Kusuda and Seme does not explicitly teach associating based 
on prior use or frequency. 

However, Bjoernsen teaches association based on said user's prior use or use frequency, 
with said contact, of said registered application agents (Bjoernsen, Fig. 10; page 1, paragraph 
6). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi, Kusuda and Seme in view of Bjoernsen in 
order to enable associating based on prior use or frequency. One would be motivated to do so 
in order to include a buddy list with popular and frequent contacts in order to allow 
collaboration between users over instant messaging services (Bjoernsen, page 1, paragraphs 2 
and 4). 

The combination of Yairi, Kusuda, Seme and Bjoernsen does not explicitly teach the 
agent automatically activated. 

However, Dickerman teaches the method further comprising the step of: automatically 
activating said registered application agents represented by said association visual cues 
whenever said contact joins said instant messaging session (Dickerman, pages 6 and 7, 
paragraphs 34-36). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi, Kusuda and Seme in view of Dickerman in 
order to enable the agent automatically activated. One would be motivated to do so in order 
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to include a buddy list with popular and frequent contacts in order to automatically activating 
an agent when a user joins in order to allow a user to invite other users to collaborate with the 
registered application (Dickerman, page 7, paragraph 36). 

30. Claims 13-20, 22, 29 and 30 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Yairi in view of Kusuda in view of Seme and further in view of Pennock et 
al. (U.S. 6,807,562). 

31. With respect to claim 13, Yairi teaches the invention described in claim 12, including a 
system for providing real-time communication over a global network in a session between 
two or more users, each of said two or more users using a device communicatively coupled 
to the global network (Yairi, page 2, paragraph 23), said system comprising: a client 
messaging application which runs on each of a plurality of user devices, at least two of said 
user devices including a user device screen, said client messaging application providing a 
user interface displayed on each of said user device screens (Yairi, page 2, paragraph 23), 
said user interface comprising: a message entry window for a user to enter data (Yairi, Fig. 
8B); a communication window for displaying messages entered in said session (Yairi, Fig. 
8B); and a selection window for accessing one or more application agents, associated to an 
external application (Yairi, pages 1-2, paragraph 9 and page 4, paragraph 38); thus providing 
an enriched communication session beyond simple, replicated text message content (Yairi, 
pages 1-2, paragraph 9 and page 4, paragraphs 38-39; GPS, phonebook, calendar, web 



Application/Control Number: 10/680,032 Page 24 

Art Unit: 2446 

browser, email, stock ticker information); wherein when one of said one or more application 
agents is activated, said external application represented by said one or more activated 
application agents (Yairi, page 2, paragraph 10; page 4, paragraph 33; page 5, paragraph 40; 
the web services). 

Yairi does not explicitly teach one or more activated application agents is displayed on 
said user device screens in conjunction with said client messaging application. 

However, Kusuda teaches one or more activated application agents is activated to run in 
conjunction with said client messaging application such that said two or more users in said 
session can utilize said external application without leaving said session (Kusuda, page 1, 
paragraph 7; page 2, paragraph 18-21). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify Yairi in view of Kusuda in order to enable one or more activated 
application agents is displayed on said user device screens in conjunction with said client 
messaging application. One would be motivated to do so in order to provide collaboration 
between two users on one external application (Kusuda, page 1, paragraphs 10 and 11). 

The combination of Yairi and Kusuda does not explicitly teach translating content as it is 
typed into said message entry window. 

However, Seme teaches a connection to said translation service application, said 
translation service application capable of translating content as it is typed into said message 
entry window (Seme, page 3, paragraphs 26, 28 and 29). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi and Kusuda in view of Seme in order to enable 
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translating content as it is typed into said message entry window. One would be motivated to 
do so in order to instantly translate real-time messages such that multilingual communication 
is achieved between one or more computing devices without compromising the real-time user 
experience (Seme, page 1, paragraph 5). 

The combination of Yairi, Kusuda and Seme does not explicitly teach registering by 
dragging. 

However, Pennock teaches wherein said user can register said unregistered application 
agent by dragging a symbol representative of said unregistered application agent from said 
list of unregistered application agents to said list of registered application agents (Pennock, 
Fig. 6; col. 6, lines 6-32 and col. 8, lines 44-54). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi, Kusuda and Seme in view of Pennock in order 
to enable registering by dragging. One would be motivated to do so in order to include 
register an application agent by dragging a symbol representative of said application agent in 
order to allow a user to select and register people to join a collaboration session (Pennock, 
col. 8, lines 44-67). 



32. With respect to claim 14, the combination of Yairi, Kusuda, Seme and Pennock teaches 
the invention described in claim 13, including wherein said symbol is an icon or a title 
(Pennock, Fig. 6; col. 6, lines 6-32). 
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33. With respect to claim 15, the combination of Yairi, Kusuda, Seme and Pennock teaches 
the invention described in claim 12, including the system wherein said user can unregister a 
registered application agent by dragging a symbol representative of said registered 
application agent from said list of registered application agents to said list of unregistered 
application agents (Pennock, Fig. 6; col. 6, lines 6-32 and col. 8, lines 44-54). 

34. With respect to claim 16, the combination of Yairi, Kusuda, Seme and Pennock teaches 
the invention described in claim 12, including the system wherein said user can register an 
unregistered application agent by applying one or more mouse-clicking commands (Pennock, 
col. 8, lines 44-54). 

35. With respect to claim 17, the combination of Yairi, Kusuda, Seme and Pennock teaches 
the invention described in claim 12, including the system wherein the user can activate a 
registered application agent by dragging a symbol representative of said registered 
application agent from said selection window to said communication window (Pennock, col. 
8, lines 44-54). 

36. With respect to claim 18, the combination of Yairi, Kusuda, Seme and Pemiock teaches 
the invention described in claim 17, including the system wherein said symbol is an icon or a 
title (Pennock, Fig. 6; col. 6, lines 6-32). 
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37. With respect to claim 19, the combination of Yairi, Kusuda, Seme and Pennock teaches 
the invention described in claim 12, including the system wherein said user can activate a 
registered application agent by applying one or more mouse-clicking commands (Pennock, 
Fig. 6; col. 6, lines 6-32). 

38. With respect to claim 20, the combination of Yairi, Kusuda, Seme and Pennock teaches 
the invention described in claim 12, including the system wherein said user can activate a 
registered application agent from a local application (Pennock, col. 6, lines 6-32 and col. 2, 
lines 20-28; gaming utility application). 

39. With respect to claim 22, the combination of Yairi, Kusuda, Seme and Pennock teaches 
the invention described in claim 12, including the system wherein said list of unregistered 
application agents is automatically updated by said client messaging application (Pennock, 
col. 2, lines 36-44). 

40. With respect to claim 29, the combination of Yairi, Kusuda, Seme and Pennock teaches 
the invention described in claim 25, including registering an unregistered application agent 
by dragging a symbol representative of said unregistered application agent from a list of 
unregistered application agents to said list of registered application agents, wherein each of 
said unregistered application agents is available to be registered with said client messaging 
application (Pennock, Fig. 6; col. 6, lines 6-32, and col. 8, lines 44-54); and unregistering a 
registered application agent by dragging a symbol representative of said registered 
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application agent from said list of registered application agents to said list of unregistered 
application agents (Pennock, Fig. 6; col. 6, lines 6-32 and col. 8, lines 44-54). 

41. With respect to claim 30, the combination of Yairi, Kusuda, Seme and Pennock teaches 
the invention described in claim 25, including the method further comprising the steps of: 
registering an unregistered application agent by applying a number of mouse-clicking 
commands (Pennock, Fig. 6; col. 6, lines 6-32 and col. 8, lines 44-54); and unregistering a 
registered application agent by applying a number of mouse-clicking commands (Pennock, 
col. 16, lines 10-22). 



42. Claim 21 is rejected under 35 U.S.C. 103(a) as being unpatentable over Yairi in view 
of Kusuda in view of Seme in view of Pennock and further in view of Bjoernsen. 

43. With respect to claim 21, Yairi teaches the invention described in claim 13, including a 
system for providing real-time communication over a global network in a session between 
two or more users, each of said two or more users using a device communicatively coupled 
to the global network (Yairi, page 2, paragraph 23), said system comprising: a client 
messaging application which runs on each of a plurality of user devices, at least two of said 
user devices including a user device screen, said client messaging application providing a 
user interface displayed on each of said user device screens (Yairi, page 2, paragraph 23), 
said user interface comprising: a message entry window for a user to enter data (Yairi, Fig. 
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8B); a communication window for displaying messages entered in said session (Yairi, Fig. 
8B); and a selection window for accessing one or more application agents, associated to an 
external application (Yairi, pages 1-2, paragraph 9 and page 4, paragraph 38); thus providing 
an enriched communication session beyond simple, replicated text message content (Yairi, 
pages 1-2, paragraph 9 and page 4, paragraphs 38-39; GPS, phonebook, calendar, web 
browser, email, stock ticker information); wherein when one of said one or more application 
agents is activated, said external application represented by said one or more activated 
application agents (Yairi, page 2, paragraph 10; page 4, paragraph 33; page 5, paragraph 40; 
the web services). 

Yairi does not explicitly teach one or more activated application agents is displayed on 
said user device screens in conjunction with said client messaging application. 

However, Kusuda teaches one or more activated application agents is activated to run in 
conjunction with said client messaging application such that said two or more users in said 
session can utilize said external application without leaving said session (Kusuda, page 1, 
paragraph 7; page 2, paragraph 18-21). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify Yairi in view of Kusuda in order to enable one or more activated 
application agents is displayed on said user device screens in conjunction with said client 
messaging application. One would be motivated to do so in order to provide collaboration 
between two users on one external application (Kusuda, page 1, paragraphs 10 and 11). 

The combination of Yairi and Kusuda does not explicitly teach translating content as it is 
typed into said message entry window. 
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However, Seme teaches a connection to said translation service application, said 
translation service application capable of translating content as it is typed into said message 
entry window (Seme, page 3, paragraphs 26, 28 and 29). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi and Kusuda in view of Seme in order to enable 
translating content as it is typed into said message entry window. One would be motivated to 
do so in order to instantly translate real-time messages such that multilingual communication 
is achieved between one or more computing devices without compromising the real-time user 
experience (Seme, page 1, paragraph 5). 

The combination of Yairi, Kusuda and Seme does not explicitly teach registering by 
dragging. 

However, Pennock teaches wherein said user can register said unregistered application 
agent by dragging a symbol representative of said unregistered application agent from said 
list of unregistered application agents to said list of registered application agents (Pennock, 
Fig. 6; col. 6, lines 6-32 and col. 8, lines 44-54). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi, Kusuda and Seme in view of Pennock in order 
to enable registering by dragging. One would be motivated to do so in order to include 
register an application agent by dragging a symbol representative of said application agent in 
order to allow a user to select and register people to join a collaboration session (Pennock, 
col. 8, lines 44-67) and the system further comprising: means for associating an alias of said 
symbol to a contact in said user's contact list (Pennock, col. 7, lines 50-62). 
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The combination of Yairi, Kusuda, Seme and Pennock does not explicitly teach alias 
symbols. 

However, Bjoernsen teaches an association based on said user's prior use of said 
application agents with said contact (Bjoernsen, Fig. 10; page 1, paragraph 6) and an 
association based on said user's prior use with said contact, of said application agent 
represented by said symbol (Bjoernsen, Fig. 10; page 1, paragraph 6). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi, Kusuda and Seme in view of Bjoernsen in 
order to enable associating contacts based on prior use. One would be motivated to do so in 
order to include a buddy list with popular and frequent contacts in order to allow 
collaboration between users over instant messaging services (Bjoernsen, page 1, paragraphs 2 
and 4). 

44. Claims 23 and 24 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Yairi in view of Kusuda in view of Seme and further in view of Dickerman. 

45. With respect to claim 23, Yairi teaches the invention described in claim 1, including a 
system for providing real-time communication over a global network in a session between 
two or more users, each of said two or more users using a device communicatively coupled 
to the global network (Yairi, page 2, paragraph 23), said system comprising: a client 
messaging application which runs on each of a plurality of user devices, at least two of said 
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user devices including a user device screen, said client messaging application providing a 
user interface displayed on each of said user device screens (Yairi, page 2, paragraph 23), 
said user interface comprising: a message entry window for a user to enter data (Yairi, Fig. 
8B); a communication window for displaying messages entered in said session (Yairi, Fig. 
8B); and a selection window for accessing one or more application agents, associated to an 
external application (Yairi, pages 1-2, paragraph 9 and page 4, paragraph 38); thus providing 
an enriched communication session beyond simple, replicated text message content (Yairi, 
pages 1-2, paragraph 9 and page 4, paragraphs 38-39; GPS, phonebook, calendar, web 
browser, email, stock ticker information); wherein when one of said one or more application 
agents is activated, said external application represented by said one or more activated 
application agents (Yairi, page 2, paragraph 10; page 4, paragraph 33; page 5, paragraph 40; 
the web services). 

Yairi does not explicitly teach one or more activated application agents is displayed on 
said user device screens in conjunction with said client messaging application. 

However, Kusuda teaches one or more activated application agents is activated to run in 
conjunction with said client messaging application such that said two or more users in said 
session can utilize said external application without leaving said session (Kusuda, page 1, 
paragraph 7; page 2, paragraph 18-21). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify Yairi in view of Kusuda in order to enable one or more activated 
application agents is displayed on said user device screens in conjunction with said client 
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messaging application. One would be motivated to do so in order to provide collaboration 
between two users on one external application (Kusuda, page 1, paragraphs 10 and 11). 

The combination of Yairi and Kusuda does not explicitly teach translating content as it is 
typed into said message entry window. 

However, Seme teaches a connection to said translation service application, said 
translation service application capable of translating content as it is typed into said message 
entry window (Seme, page 3, paragraphs 26, 28 and 29). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi and Kusuda in view of Seme in order to enable 
translating content as it is typed into said message entry window. One would be motivated to 
do so in order to instantly translate real-time messages such that multilingual communication 
is achieved between one or more computing devices without compromising the real-time user 
experience (Seme, page 1, paragraph 5). 

The combination of Yairi, Kusuda and Seme does not explicitly teach activating agents 
upon inviting another user. 

However, Dickerman teaches the system further comprising: wherein a user invites 
another user to activate one of said application agents in said session (Dickerman, page 7, 
paragraph 41). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi, Kusuda and Seme in view of Dickerman in 
order to enable activating agents upon inviting another user. One would be motivated to do 
so in order to include a buddy list with popular and frequent contacts in order to 
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automatically activating an agent when a user joins in order to allow a user to invite other 
users to collaborate with the registered application (Dickerman, page 7, paragraph 36). 

46. With respect to claim 24, the combination of Yairi, Kusuda, Seme and Dickerman 
teaches the invention described in claim 1 , including the system wherein said user interface 
further comprises: a system for selection by a user to initiate synchronous sharing of a 
service represented by an activated application agent (Dickerman, page 2, paragraph 24 and 
26, and page 7, paragraph 36). 

47. Claims 34-36, 39 and 44 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Yairi in view of Kusuda and further in view of Audu et al. (U.S. 2005/0049879). 

48. With respect to claim 34, Yairi teaches a system for providing real-time communication 
over a global network in a session between two or more users, each of said two or more users 
using a device communicatively coupled to the global network (Yairi, page 2, paragraph 23), 
said system comprising: a client messaging application which runs on each of a plurality of 
user devices, at least two of said user devices including a user device screen, said client 
messaging application providing a user interface displayed on each of said user device 
screens (Yairi, page 2, paragraph 23), said user interface comprising: a message entry 
window for a user to enter data (Yairi, Fig. 8B); a communication window for displaying 
messages entered in said session (Yairi, Fig. 8B); and a selection window for accessing one 



Application/Control Number: 1 0/680,032 Page 3 5 

Art Unit: 2446 

or more application agents, a connection to said application(Yairi, pages 1-2, paragraph 9 and 
page 4, paragraph 38), said application capable of providing an enriched communication 
session beyond simple, replicated text message content (Yairi, pages 1-2, paragraph 9 and 
page 4, paragraphs 38-39; GPS, phonebook, calendar, web browser, email, stock ticker 
information). 

Yairi does not explicitly teach one or more activated application agents is displayed on 
said user device screens in conjunction with said client messaging application. 

However, Kusuda teaches wherein one or more activated application agents is activated, 
said application represented by said one or more activated application agents is activated to 
run in conjunction with said client messaging application such that said two or more users in 
said session can utilize said external application without leaving said session (Kusuda, page 
1, paragraph 7; page 2, paragraph 18-21). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify Yairi in view of Kusuda in order to enable one or more activated 
application agents is displayed on said user device screens in conjunction with said client 
messaging application. One would be motivated to do so in order to provide collaboration 
between two users on one external application (Kusuda, page 1, paragraphs 10 and 11). 

The combination of Yairi and Kusuda does not explicitly teach translating text to speech. 

However, Audu teaches said application agent associated with a speech synthesis 
application running on at least one device from among the plurality of devices (Audu, pages 
2 and 3, paragraphs 31-37), wherein said speech synthesis application is part of a system for 
speech synthesis, the system for speech synthesis comprising: a speech enabled input device 
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for recognizing audible human speech and translating said human speech into textual data 
(Audu, page 3, paragraphs 41 and 42); a screen reader for recognizing textual data in one or 
more applications running on said user devices and translating said textual data into human- 
recognizable auditory signals; and an audio output for outputting said human-recognizable 
auditory signals (Audu, pages 3 and 4, paragraphs 45 and 46). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi and Kusuda in view of Audu in order to enable 
translating text to speech. One would be motivated to do so in order to have a communication 
device that would tie all of this technology together in a seamless manner such that a user 
could select whether they want to use voice communications or text communications to 
communicate in real time through a communications network with a user of a remote 
communication device that can be a voice-and-text capable communication device (Audu, 
page 1, paragraph 9, and page 2, paragraph 24). 

49. With respect to claim 35, the combination of Yairi, Kusuda and Audu teaches the 
invention described in claim 34, including the system wherein said speech synthesis 
application is either a local application in any of said user's device (Audu, pages 2 and 3, 
paragraphs 31-37) or a service on the global network. 

50. With respect to claim 36, the combination of Yairi, Kusuda and Audu teaches the 
invention described in claim 34, including the system wherein said activated application 
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agent addresses said speech synthesis application on an other side of said session via the 
user's message protocol (Audu, pages 3 and 4, paragraph 46). 

51. With respect to claim 39, the combination of Yairi, Kusuda and Audu teaches the 
invention described in claim 34, including the system wherein said selection window 
displays any of: a list of unregistered application agents, each of said unregistered application 
agents being available to be registered with said client messaging application (Yairi, Figs. 4 
and 8A; page 4, paragraph 39); and a list of registered application agents, each of said 
registered application agents being immediately available to be activated by a user (Yairi, 
Figs. 4 and 8 A; page 4, paragraph 39). 

52. With respect to claim 44, Yairi teaches a method for incorporating external resources into 
an instant messaging session supported by an instant messaging system (Yairi, page 2, 
paragraph 23), said instant messaging system comprising a client messaging application 
which runs on devices communicatively coupled to the Internet (Yairi, page 2, paragraph 23), 
comprising the steps of: providing through said client messaging application a user interface 
displayed on each device's screen from which a user communicates with another user (Yairi, 
page 2, paragraph 23), said user interface comprising a message entry window for said user 
to enter data (Yairi, Fig. 8B), a communication window for displaying messages entered in 
said instant messaging session (Yairi, Fig. 8B), and a selection window for accessing one or 
more application agents, each of said application agents being associated with an external 
application (Yairi, pages 1 and 2, paragraph 9, and page 4, paragraph 38), the method further 
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comprising the steps of: activating an application agent from a list of available application 
agents, wherein each of said application agents is available to be activated by said user 
(Yairi, page 2, paragraph 10 and page 4, paragraph 33), thereby activating the external 
application to which said activated application agent is associated, said external application 
capable of providing an enriched communication session beyond simple, replicated text 
message content (Yairi, pages 1 and 2, paragraph 9, and page 4, paragraphs 38 and 39; GPS, 
phonebook, calendar, web browser, email, stock ticker information). 
Yairi does not explicitly teach sharing said external application. 

However, Kusuda teaches sharing said external application between at least two users in 
said instant messaging session (Kusuda, page 1, paragraph 7, and page 2, paragraphs 18-21). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify Yairi in view of Kusuda in order to enable sharing said external 
application. One would be motivated to do so in order to provide collaboration between two 
users on one external application (Kusuda, page 1, paragraphs 10 and 11). 

The combination of Yairi and Kusuda does not explicitly teach translating text to speech. 

However, Audu teaches said application agent associated with a speech synthesis 
application running on at least one device from among the devices (Audu, pages 2 and 3, 
paragraphs 31-37), wherein said speech synthesis application is part of a system for speech 
synthesis, the system for speech synthesis comprising: a speech enabled input device for 
recognizing audible human speech and translating said human speech into textual data 
(Audu, page 3, paragraphs 41 and 42); a screen reader for recognizing textual data in one or 
more applications running on said user devices and translating said textual data into human- 
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recognizable auditory signals; and an audio output for outputting said human-recognizable 
auditory signals (Audu, pages 3 and 4, paragraphs 45 and 46). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi and Kusuda in view of Audu in order to enable 
translating text to speech. One would be motivated to do so in order to have a communication 
device that would tie all of this technology together in a seamless manner such that a user 
could select whether they want to use voice communications or text communications to 
communicate in real time through a communications network with a user of a remote 
communication device that can be a voice-and-text capable communication device (Audu, 
page 1, paragraph 9, and page 2, paragraph 24). 

53. Claim 37 is rejected under 35 U.S.C. 103(a) as being unpatentable over Yairi in view of 
Kusuda in view of Audu and further in view of Bjoernsen. 

54. With respect to claim 37, Yairi teaches the invention described in claim 34, including a 
system for providing real-time communication over a global network in a session between 
two or more users, each of said two or more users using a device communicatively coupled 
to the global network (Yairi, page 2, paragraph 23), said system comprising: a client 
messaging application which runs on each of a plurality of user devices, at least two of said 
user devices including a user device screen, said client messaging application providing a 
user interface displayed on each of said user device screens (Yairi, page 2, paragraph 23), 
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said user interface comprising: a message entry window for a user to enter data (Yairi, Fig. 
8B); a communication window for displaying messages entered in said session (Yairi, Fig. 
8B); and a selection window for accessing one or more application agents, a connection to 
said application(Yairi, pages 1-2, paragraph 9 and page 4, paragraph 38), said application 
capable of providing an enriched communication session beyond simple, replicated text 
message content (Yairi, pages 1-2, paragraph 9 and page 4, paragraphs 38-39; GPS, 
phonebook, calendar, web browser, email, stock ticker information) and wherein said 
selection window for accessing one or more application agents further comprises: one or 
more distinct visual cues, each of which being representative of one or more said application 
agents (Yairi, Fig. 4 and 8a; page 4, paragraph 39); and said one or more distinct visual cues 
shown in association with contact in a contact list of a user (Yairi, Fig. 4 and 8a; page 4, 
paragraph 39). 

Yairi does not explicitly teach one or more activated application agents is displayed on 
said user device screens in conjunction with said client messaging application. 

However, Kusuda teaches wherein one or more activated application agents is activated, 
said application represented by said one or more activated application agents is activated to 
run in conjunction with said client messaging application such that said two or more users in 
said session can utilize said external application without leaving said session (Kusuda, page 
1, paragraph 7; page 2, paragraph 18-21). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify Yairi in view of Kusuda in order to enable one or more activated 
application agents is displayed on said user device screens in conjunction with said client 
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messaging application. One would be motivated to do so in order to provide collaboration 
between two users on one external application (Kusuda, page 1, paragraphs 10 and 11). 

The combination of Yairi and Kusuda does not explicitly teach translating text to speech. 

However, Audu teaches said application agent associated with a speech synthesis 
application running on at least one device from among the plurality of devices (Audu, pages 
2 and 3, paragraphs 31-37), wherein said speech synthesis application is part of a system for 
speech synthesis, the system for speech synthesis comprising: a speech enabled input device 
for recognizing audible human speech and translating said human speech into textual data 
(Audu, page 3, paragraphs 41 and 42); a screen reader for recognizing textual data in one or 
more applications running on said user devices and translating said textual data into human- 
recognizable auditory signals; and an audio output for outputting said human-recognizable 
auditory signals (Audu, pages 3 and 4, paragraphs 45 and 46). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi and Kusuda in view of Audu in order to enable 
translating text to speech. One would be motivated to do so in order to have a communication 
device that would tie all of this technology together in a seamless manner such that a user 
could select whether they want to use voice communications or text communications to 
communicate in real time through a communications network with a user of a remote 
communication device that can be a voice-and-text capable communication device (Audu, 
page 1, paragraph 9, and page 2, paragraph 24). 

The combination of Yairi, Kusuda and Audu does not explicitly teach associating 
contacts based on prior use. 
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However, Bjoernsen teaches an association based on said user's prior use of said 
application agents with said contact (Bjoernsen, Fig. 10; page 1, paragraph 6). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi, Kusuda and Audu in view of Bjoernsen in 
order to enable associating contacts based on prior use. One would be motivated to do so in 
order to include a buddy list with popular and frequent contacts in order to allow 
collaboration between users over instant messaging services (Bjoernsen, page 1, paragraphs 2 
and 4). 

55. Claim 38 is rejected under 35 U.S.C. 103(a) as being unpatentable over Yairi in view 
of Kusuda in view of Audu in view of Bjoernsen and further in view of Dickerman. 

56. With respect to claim 38, Yairi teaches the invention described in claim 37, including a 
system for providing real-time communication over a global network in a session between 
two or more users, each of said two or more users using a device communicatively coupled 
to the global network (Yairi, page 2, paragraph 23), said system comprising: a client 
messaging application which runs on each of a plurality of user devices, at least two of said 
user devices including a user device screen, said client messaging application providing a 
user interface displayed on each of said user device screens (Yairi, page 2, paragraph 23), 
said user interface comprising: a message entry window for a user to enter data (Yairi, Fig. 
8B); a communication window for displaying messages entered in said session (Yairi, Fig. 
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8B); and a selection window for accessing one or more application agents, a connection to 
said application(Yairi, pages 1-2, paragraph 9 and page 4, paragraph 38), said application 
capable of providing an enriched communication session beyond simple, replicated text 
message content (Yairi, pages 1-2, paragraph 9 and page 4, paragraphs 38-39; GPS, 
phonebook, calendar, web browser, email, stock ticker information) and wherein said 
selection window for accessing one or more application agents further comprises: one or 
more distinct visual cues, each of which being representative of one or more said application 
agents (Yairi, Fig. 4 and 8a; page 4, paragraph 39); and said one or more distinct visual cues 
shown in association with contact in a contact list of a user (Yairi, Fig. 4 and 8a; page 4, 
paragraph 39). 

Yairi does not explicitly teach one or more activated application agents is displayed on 
said user device screens in conjunction with said client messaging application. 

However, Kusuda teaches wherein one or more activated application agents is activated, 
said application represented by said one or more activated application agents is activated to 
run in conjunction with said client messaging application such that said two or more users in 
said session can utilize said external application without leaving said session (Kusuda, page 
1, paragraph 7; page 2, paragraph 18-21). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify Yairi in view of Kusuda in order to enable one or more activated 
application agents is displayed on said user device screens in conjunction with said client 
messaging application. One would be motivated to do so in order to provide collaboration 
between two users on one external application (Kusuda, page 1, paragraphs 10 and 11). 
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The combination of Yairi and Kusuda does not explicitly teach translating text to speech. 

However, Audu teaches said application agent associated with a speech synthesis 
application running on at least one device from among the plurality of devices (Audu, pages 
2 and 3, paragraphs 31-37), wherein said speech synthesis application is part of a system for 
speech synthesis, the system for speech synthesis comprising: a speech enabled input device 
for recognizing audible human speech and translating said human speech into textual data 
(Audu, page 3, paragraphs 41 and 42); a screen reader for recognizing textual data in one or 
more applications running on said user devices and translating said textual data into human- 
recognizable auditory signals; and an audio output for outputting said human-recognizable 
auditory signals (Audu, pages 3 and 4, paragraphs 45 and 46). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi and Kusuda in view of Audu in order to enable 
translating text to speech. One would be motivated to do so in order to have a communication 
device that would tie all of this technology together in a seamless manner such that a user 
could select whether they want to use voice communications or text communications to 
communicate in real time through a communications network with a user of a remote 
communication device that can be a voice-and-text capable communication device (Audu, 
page 1, paragraph 9, and page 2, paragraph 24). 

The combination of Yairi, Kusuda and Audu does not explicitly teach associating 
contacts based on prior use. 

However, Bjoernsen teaches an association based on said user's prior use of said 
application agents with said contact (Bjoernsen, Fig. 10; page 1, paragraph 6). 
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It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi, Kusuda and Audu in view of Bjoernsen in 
order to enable associating contacts based on prior use. One would be motivated to do so in 
order to include a buddy list with popular and frequent contacts in order to allow 
collaboration between users over instant messaging services (Bjoernsen, page 1, paragraphs 2 
and 4). 

The combination of Yairi, Kusuda, Audu and Bjoernsen does not explicitly teach the 
agent automatically activated. 

However, Dickerman teaches wherein any of said application agents can be registered as 
a contact in said contact list of said user to create registered application agents (Dickerman, 
page 7, paragraph 36). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi, Kusuda, Audu and Bjoernsen in view of 
Dickerman in order to enable the agent automatically activated. One would be motivated to 
do so in order to include a buddy list with popular and frequent contacts in order to 
automatically activating an agent when a user joins in order to allow a user to invite other 
users to collaborate with the registered application (Dickerman, page 7, paragraph 36). 

57. Claims 40 and 41 are rejected under 35 U.S.C. 103(a) as being unpatentable over 

Yairi in view of Kusuda in view of Audu in view of Pennock. 
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58. With respect to claim 40, Yairi teaches the invention described in claim 39, including a a 
system for providing real-time communication over a global network in a session between 
two or more users, each of said two or more users using a device communicatively coupled 
to the global network (Yairi, page 2, paragraph 23), said system comprising: a client 
messaging application which runs on each of a plurality of user devices, at least two of said 
user devices including a user device screen, said client messaging application providing a 
user interface displayed on each of said user device screens (Yairi, page 2, paragraph 23), 
said user interface comprising: a message entry window for a user to enter data (Yairi, Fig. 
8B); a communication window for displaying messages entered in said session (Yairi, Fig. 
8B); and a selection window for accessing one or more application agents, a connection to 
said application( Yairi, pages 1-2, paragraph 9 and page 4, paragraph 38), said application 
capable of providing an enriched communication session beyond simple, replicated text 
message content (Yairi, pages 1-2, paragraph 9 and page 4, paragraphs 38-39; GPS, 
phonebook, calendar, web browser, email, stock ticker information) and the system wherein 
said selection window displays any of: a list of unregistered application agents, each of said 
unregistered application agents being available to be registered with said client messaging 
application (Yairi, Figs. 4 and 8 A; page 4, paragraph 39); and a list of registered application 
agents, each of said registered application agents being immediately available to be activated 
by a user (Yairi, Figs. 4 and 8 A; page 4, paragraph 39). 

Yairi does not explicitly teach one or more activated application agents is displayed on 
said user device screens in conjunction with said client messaging application. 
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However, Kusuda teaches wherein one or more activated application agents is activated, 
said application represented by said one or more activated application agents is activated to 
run in conjunction with said client messaging application such that said two or more users in 
said session can utilize said external application without leaving said session (Kusuda, page 
1, paragraph 7; page 2, paragraph 18-21). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify Yairi in view of Kusuda in order to enable one or more activated 
application agents is displayed on said user device screens in conjunction with said client 
messaging application. One would be motivated to do so in order to provide collaboration 
between two users on one external application (Kusuda, page 1, paragraphs 10 and 11). 

The combination of Yairi and Kusuda does not explicitly teach translating text to speech. 

However, Audu teaches said application agent associated with a speech synthesis 
application running on at least one device from among the plurality of devices (Audu, pages 
2 and 3, paragraphs 31-37), wherein said speech synthesis application is part of a system for 
speech synthesis, the system for speech synthesis comprising: a speech enabled input device 
for recognizing audible human speech and translating said human speech into textual data 
(Audu, page 3, paragraphs 41 and 42); a screen reader for recognizing textual data in one or 
more applications running on said user devices and translating said textual data into human- 
recognizable auditory signals; and an audio output for outputting said human-recognizable 
auditory signals (Audu, pages 3 and 4, paragraphs 45 and 46). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi and Kusuda in view of Audu in order to enable 
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translating text to speech. One would be motivated to do so in order to have a communication 
device that would tie all of this technology together in a seamless manner such that a user 
could select whether they want to use voice communications or text communications to 
communicate in real time through a communications network with a user of a remote 
communication device that can be a voice-and-text capable communication device (Audu, 
page 1, paragraph 9, and page 2, paragraph 24). 

The combination of Yairi, Kusuda and Audu does not explicitly teach registering by 
dragging. 

However, Pennock teaches wherein said user can register said unregistered application 
agent by dragging a symbol representative of said unregistered application agent from said 
list of unregistered application agents to said list of registered application agents (Pennock, 
Fig. 6; col. 6, lines 6-32 and col. 8, lines 44-54). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi, Kusuda and Audu in view of Pennock in order 
to enable registering by dragging. One would be motivated to do so in order to include 
register an application agent by dragging a symbol representative of said application agent in 
order to allow a user to select and register people to join a collaboration session (Pennock, 
col. 8, lines 44-67). 



59. With respect to claim 41, the combination of Yairi, Kusuda, Audu and Pennock teaches 
the invention described in claim 40, including wherein said symbol is an icon or a title 
(Pennock, Fig. 6; col. 6, lines 6-32). 
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60. Claims 42 and 43 are rejected under 35 U.S.C. 103(a) as being unpatentable over Yairi in 
view of Kusuda in view of Audu and further in view of Dickerman. 

61. With respect to claim 42, Yairi teaches the invention described in claim 34, including a 
system for providing real-time communication over a global network in a session between 
two or more users, each of said two or more users using a device communicatively coupled 
to the global network (Yairi, page 2, paragraph 23), said system comprising: a client 
messaging application which runs on each of a plurality of user devices, at least two of said 
user devices including a user device screen, said client messaging application providing a 
user interface displayed on each of said user device screens (Yairi, page 2, paragraph 23), 
said user interface comprising: a message entry window for a user to enter data (Yairi, Fig. 
8B); a communication window for displaying messages entered in said session (Yairi, Fig. 
8B); and a selection window for accessing one or more application agents, a connection to 
said application(Yairi, pages 1-2, paragraph 9 and page 4, paragraph 38), said application 
capable of providing an enriched communication session beyond simple, replicated text 
message content (Yairi, pages 1-2, paragraph 9 and page 4, paragraphs 38-39; GPS, 
phonebook, calendar, web browser, email, stock ticker information). 

Yairi does not explicitly teach one or more activated application agents is displayed on 
said user device screens in conjunction with said client messaging application. 

However, Kusuda teaches wherein one or more activated application agents is activated, 
said application represented by said one or more activated application agents is activated to 
run in conjunction with said client messaging application such that said two or more users in 
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said session can utilize said external application without leaving said session (Kusuda, page 
1, paragraph 7; page 2, paragraph 18-21). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify Yairi in view of Kusuda in order to enable one or more activated 
application agents is displayed on said user device screens in conjunction with said client 
messaging application. One would be motivated to do so in order to provide collaboration 
between two users on one external application (Kusuda, page 1, paragraphs 10 and 11). 

The combination of Yairi and Kusuda does not explicitly teach translating text to speech. 

However, Audu teaches said application agent associated with a speech synthesis 
application running on at least one device from among the plurality of devices (Audu, pages 
2 and 3, paragraphs 31-37), wherein said speech synthesis application is part of a system for 
speech synthesis, the system for speech synthesis comprising: a speech enabled input device 
for recognizing audible human speech and translating said human speech into textual data 
(Audu, page 3, paragraphs 41 and 42); a screen reader for recognizing textual data in one or 
more applications running on said user devices and translating said textual data into human- 
recognizable auditory signals; and an audio output for outputting said human-recognizable 
auditory signals (Audu, pages 3 and 4, paragraphs 45 and 46). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi and Kusuda in view of Audu in order to enable 
translating text to speech. One would be motivated to do so in order to have a communication 
device that would tie all of this technology together in a seamless manner such that a user 
could select whether they want to use voice communications or text communications to 
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communicate in real time through a communications network with a user of a remote 
communication device that can be a voice-and-text capable communication device (Audu, 
page 1, paragraph 9, and page 2, paragraph 24). 

The combination of Yairi, Kusuda and Audu does not explicitly teach activating agents 
upon inviting another user. 

However, Dickerman teaches the system further comprising: wherein a user invites 
another user to activate one of said application agents in said session (Dickerman, page 7, 
paragraph 41). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi, Kusuda and Audu in view of Dickerman in 
order to enable activating agents upon inviting another user. One would be motivated to do 
so in order to include a buddy list with popular and frequent contacts in order to 
automatically activating an agent when a user joins in order to allow a user to invite other 
users to collaborate with the registered application (Dickerman, page 7, paragraph 36). 

62. With respect to claim 43, the combination of Yairi, Kusuda, Audu and Dickerman 
teaches the invention described in claim 34, including the system wherein said user interface 
further comprises: a system for selection by a user to initiate synchronous sharing of a 
service represented by an activated application agent (Dickerman, page 2, paragraph 24 and 
26, and page 7, paragraph 36). 
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63. Claim 45 is rejected under 35 U.S.C. 103(a) as being unpatentable over Yairi in view of 
Kusuda in view of Audu in view of Bjornsen and further in view of Dickerman. 

64. With respect to claim 45, Yairi teaches the invention described in claim 44, including a 
method for incorporating external resources into an instant messaging session supported by 
an instant messaging system (Yairi, page 2, paragraph 23), said instant messaging system 
comprising a client messaging application which runs on devices communicatively coupled 
to the Internet (Yairi, page 2, paragraph 23), comprising the steps of: providing through said 
client messaging application a user interface displayed on each device's screen from which a 
user communicates with another user (Yairi, page 2, paragraph 23), said user interface 
comprising a message entry window for said user to enter data (Yairi, Fig. 8B), a 
communication window for displaying messages entered in said instant messaging session 
(Yairi, Fig. 8B), and a selection window for accessing one or more application agents, each 
of said application agents being associated with an external application (Yairi, pages 1 and 2, 
paragraph 9, and page 4, paragraph 38), the method further comprising the steps of: 
activating an application agent from a list of available application agents, wherein each of 
said application agents is available to be activated by said user (Yairi, page 2, paragraph 10 
and page 4, paragraph 33), thereby activating the external application to which said activated 
application agent is associated, said external application capable of providing an enriched 
communication session beyond simple, replicated text message content (Yairi, pages 1 and 2, 
paragraph 9, and page 4, paragraphs 38 and 39; GPS, phonebook, calendar, web browser, 
email, stock ticker information); associating one or more of visual cues to a contact in a 
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contact list of a user (Yairi, Figs. 4 and 8 A; page 4, paragraph 39), each of said one or more 
visual cues representing one of said registered application agents (Yairi, Figs. 4 and 8A; page 
4, paragraph 39); and the method wherein said contact can be any of: a screen name 
representing a human contact (Yairi, Fig. 8B); a name or a visual cue representing an 
interactive service; and a name or a visual cue representing one of said registered application 
agents. 

Yairi does not explicitly teach sharing said external application. 

However, Kusuda teaches sharing said external application between at least two users in 
said instant messaging session (Kusuda, page 1, paragraph 7, and page 2, paragraphs 18-21). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify Yairi in view of Kusuda in order to enable sharing said external 
application. One would be motivated to do so in order to provide collaboration between two 
users on one external application (Kusuda, page 1, paragraphs 10 and 11). 

The combination of Yairi and Kusuda does not explicitly teach translating text to speech. 

However, Audu teaches said application agent associated with a speech synthesis 
application running on at least one device from among the devices (Audu, pages 2 and 3, 
paragraphs 31-37), wherein said speech synthesis application is part of a system for speech 
synthesis, the system for speech synthesis comprising: a speech enabled input device for 
recognizing audible human speech and translating said human speech into textual data 
(Audu, page 3, paragraphs 41 and 42); a screen reader for recognizing textual data in one or 
more applications running on said user devices and translating said textual data into human- 
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recognizable auditory signals; and an audio output for outputting said human-recognizable 
auditory signals (Audu, pages 3 and 4, paragraphs 45 and 46). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi and Kusuda in view of Audu in order to enable 
translating text to speech. One would be motivated to do so in order to have a communication 
device that would tie all of this technology together in a seamless manner such that a user 
could select whether they want to use voice communications or text communications to 
communicate in real time through a communications network with a user of a remote 
communication device that can be a voice-and-text capable communication device (Audu, 
page 1, paragraph 9, and page 2, paragraph 24). 

The combination of Yairi, Kusuda and Audu does not explicitly teach associating based 
on prior use or frequency. 

However, Bjoernsen teaches association based on said user's prior use or use frequency, 
with said contact, of said registered application agents (Bjoernsen, Fig. 10; page 1, paragraph 
6). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi, Kusuda and Audu in view of Bjoernsen in 
order to enable associating based on prior use or frequency. One would be motivated to do so 
in order to include a buddy list with popular and frequent contacts in order to allow 
collaboration between users over instant messaging services (Bjoernsen, page 1, paragraphs 2 
and 4). 
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The combination of Yairi, Kusuda, Audu and Bjoernsen does not explicitly teach the 
agent automatically activated. 

However, Dickerman teaches the method further comprising the step of: automatically 
activating said registered application agents represented by said association visual cues 
whenever said contact joins said instant messaging session (Dickerman, pages 6 and 7, 
paragraphs 34-36). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to modify the combination of Yairi, Kusuda, Audu and Bjoernsen in view of 
Dickerman in order to enable the agent automatically activated. One would be motivated to 
do so in order to include a buddy list with popular and frequent contacts in order to 
automatically activating an agent when a user joins in order to allow a user to invite other 
users to collaborate with the registered application (Dickerman, page 7, paragraph 36). 
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Conclusion 

Applicant's amendment necessitated the new ground(s) of rejection presented in this Office 
action. Accordingly, THIS ACTION IS MADE FINAL. Applicant is reminded of the 
extension of time policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE MONTHS from 
the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the 
mailing date of this final action and the advisory action is not mailed until after the end of the 
THREE-MONTH shortened statutory period, then the shortened statutory period will expire on 
the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be 
calculated from the mailing date of the advisory action. In no event, however, will the statutory 
period for reply expire later than SIX MONTHS from the mailing date of this final action. 

Any inquiry concerning this communication or earlier communications from the examiner 
should be directed to Alicia Baturay whose telephone number is (571) 272-3981. The examiner 
can normally be reached at 7:30am - 5pm, Monday - Thursday. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, 
Jeffrey Pwu can be reached on (571) 272-6798. The fax number for the organization where this 
application or proceeding is assigned is (571) 273-8300. 
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Information regarding the status of an application may be obtained from the Patent Application 
Information Retrieval (PAIR) system. Status information for published applications may be 
obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 

Alicia Baturay 
October 22, 2009 



/Jeffrey Pwu/ 

Supervisory Patent Examiner, Art Unit 2446 



